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Abstract 

The purpose of this study was to explore the present status of efforts to assess student-learning 
outcomes within the bachelor’s degree granting institutions of the campuses in one system of 
public higher education. Further, the purpose of this study was also to understand what 
challenges and criticisms academic leaders report about the call to provide learning outcome 
evidence. Themes and findings of the study suggest that accountability for student learning is a 
persistent accountability challenge within higher education institutions. 


Accountability is a persistent and commanding policy issue facing higher education leaders, 
and student learning is a predominant form of accountability evidence requested by 
stakeholders (National Governor’s Association, 1986; Thelin, 2004; Spellings Commission, 
2006). Two high level reports provide historical bookends on the persistent call for 
accountability evidence of student learning. In 1986, the National Governor’s Association 
released a report entitled Time for Results, outlining a plan to reform education in the United 
States. One of the major foci called for nationwide commitment on the part of institutional 
leaders to focus on improving educational quality and to produce evidence of student 
learning (National Governor’s Association, 1986). Twenty years later, the Spellings 
Commission (2006) released “A test of leadership: Charting the future ofU.S. higher 
education .” That report once again raised concern with student learning and called for a 
nationwide effort to re-position the United States as a world leader in education. 

Joining these calls have been efforts from lawmakers to push toward institutional 
accountability for student learning (Flaherty, 2013; Board of Governors, State University 
System of Florida, 2013). In Iowa, for instance, the legislature passed and the Governor 
signed a mandate that faculty who teach at each of the three state universities create 
assessments that provide summative information on student performance and formative 
results that can be used to generate a plan for improvement (Flaherty, 2013). Florida’s 
state universities are required to develop an ‘ Academic Learning Compact ’ for each 
undergraduate educational program, which articulates the expected outcomes, 
corresponding assessment instruments and methods, and a plan for improvement based 
on the results (State University System of Florida, 2013). 

In addition to state-level mandates, accreditation has added pressure to the expectation to 
assess and report student learning outcomes. Ewell (2001) stated that accreditation 
organizations, both regional and specialized, have a central role in pushing assessment of 
student learning outcomes to the forefront at colleges and universities to demonstrate 
institutional effectiveness. For instance, the Southern Association of Colleges and 
Schools Commission on Colleges (2012) requires that all educational programs identify a 
clearly articulated set of student learning outcomes and that such outcomes are 
accompanied by a plan for how the results will be used to guide improvement. With the 
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requirement that federal student aid is tied to regional accreditation, universities have an 
added incentive to prioritize the assessment of student learning outcomes (United States 
Department of Education, 2013). 

But despite longstanding and widespread interest, accountability for student learning 
persists as a desired, but largely unaddressed, form of accountability evidence. 
Consequently, this paper seeks to describe the current status and challenges associated 
with institutional efforts to respond to expectations for learning evidence. This 
description will not be given, however, without first considering the underlying 
contextual factors that influence these efforts. Several questions are warranted to give 
consideration to this context. 

Why are stakeholders concerned with accountability in student learning? Many of the 
students who graduate from colleges and universities are perceived to lack core 
competencies associated with an undergraduate education. A recent book entitled 
Academically Adrift by Arum and Roksa (2011) criticized colleges and universities for not 
asking enough of students, arguing that insufficient rigor has led to student incompetence 
with basic skills such as written communication and critical thinking. Examining test 
scores, survey responses, and transcripts of over 2,000 students, Arum and Roksa found 
that students do not significantly improve in these skill domains as a result of the 
undergraduate curriculum. 

In addition, employers commonly report that graduates lack basic skills to meet job- 
related demands. In 2010, the American Association of Colleges and Universities 
released a report on employer perceptions of college learning. Employers were 
interviewed about the skills needed to address challenges associated with a globalizing 
marketplace and about their perspectives on the effectiveness of institutions to prepare 
students with the skill sets to meet these challenges. Participants shared that a broad 
range of skills and a specialized understanding of a field were necessary for success in 
today’s workplaces. However, employers also reported that many graduates entered the 
workplace insufficiently prepared to meet these expectations and that college and 
university leaders should address these skill deficiencies (American Association of 
Colleges & Universities, 2010). 

Aside from perceived deficits in the competencies of college graduates, many of 
higher education’s key stakeholders view institutions as incapable of or disinterested 
in sufficiently addressing accountability and performance expectations (Lane, 2007; 
Bogue & Hall, 2012). Bogue and Hall (2012) surveyed corporate executives, 
legislators, and academic leaders in five different states to gather and compare major 
stakeholder perspectives on higher education accountability. Bogue and Hall found 
that corporate and legislative leaders perceived that institutions will ‘use cosmetic 
changes to avoid disclosing unflattering information on performance.’ Other evidence 
suggests that resistance to external expectations for change in institutional 
accountability for learning may insufficiently characterize the issue of institutional 
response to assessment. Lane (2007) wrote, for instance, that a key factor influencing 
resistance, where truth to this notion may be observed, can be due to the uncertainty 
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surrounding the change proposition. What is it, specifically, that stakeholders want as 
evidence for student learning, it is grounded in consensus, and is there a clear 
connection between what is expected and what is taught on college and university 
campuses? At the heart of accountability for student learning may, in part, be the 
uncertainty about how to navigate tension between an enterprise that is valued for its 
autonomy in academic mission and evaluated by its duty to demonstrate performance 
within a democratic social and political fabric. Further, a lack of clarity is present on 
the extent to which institutions are truly resistant to change or whether this perception 
is more closely associated with some stakeholder dissatisfaction with institutional 
mission priorities. 

What has been the response from colleges and universities thus far? Despite the 
complexity of the task to be accountable for student learning, some institutions have taken 
initiative to provide transparent, comparable data to stakeholders (Association of Public and 
Land Grant Universities, 2013; The Lumina Foundation, 2013). The Association for 
Public and Land Grant Universities’ Voluntary System of Accountability, a resource in 
which many colleges and universities elect to participate, launched a pilot program in 
2012 that built in evidence of student learning into the profile of key performance criteria 
on student outcomes and experience indicators. Collected through the pilot project on 
student learning were scores on standardized exams designed to measure outcomes 
commonly associated with the general education curriculum (Association of Public and 
Land Grant Universities, 2012). 

Non-profit organizations have also partnered with colleges and universities to address the 
call for assessment of student learning. For example, the Lumina Foundation (2013) has 
developed the Degree Qualifications Profile (DQP), which is a resource that seeks to define 
competencies that a graduate should be expected to know upon completion of an 
associate’s, baccalaureate, or master’s degree - though a corresponding, comprehensive 
plan for how universities can assess and report on such student learning is missing. In 
partnership with Lumina, the National Institute for Learning Outcomes Assessment, located 
at the University of Illinois, Urbana-Champaign, has been working to track institutional use 
of the DQP and found that hundreds of institutions across 46 states are using this framework 
for assessment (National Institute for Learning Outcomes Assessment, 2013). Through 
these initiatives, it is evident that many colleges and universities are working to address 
concerns related to student learning. The impact of these efforts to inform the public and to 
guide improvement is yet to be determined. 

What makes being accountable for student learning difficult? Evidence suggests there 
is inadequate consensus about what forms of learning evidence are considered desirable 
among major stakeholders (Bogue & Hall, 2012). In Bogue and Hall’s five-state study 
on corporate, political, and academic leader views on accountability, for instance, there 
was wide disagreement between the three groups on the desirability of various student 
learning outcomes as an indicator of performance. Business and state political 
stakeholders differed from college and university presidents, provosts, and faculty on the 
desirability of student knowledge of democratic culture and heritage as an indicator of 
performance. The ability to pursue knowledge through different modes of inquiry was 
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also viewed as less desirable among business and political stakeholders as compared to 
academic leaders. The challenge to identify what should be measured may provide 
difficulty for college and university leaders to effectively demonstrate accountability to 
stakeholders whose viewpoints vary on the desirable outcomes expected of graduates. 

A study conducted by the National Association for College Admissions Counseling 
(2008) concluded that one-third of students entering a two- or four-year college or 
university in the United States will transfer to at least one other institution during their 
time in college. Given overlap in broad competency areas taught at each institution to 
which mobile learners attend, it may be difficult to target the effectiveness of learning 
interventions specific to one institution or another. In addition, college is a place where 
students are presented with multiple opportunities to develop essential competencies 
across the curriculum. It can also be difficult to ascertain with confidence the extent to 
which assessment results offer a true reflection of students’ learning performance in 
relation to a particular institution or curriculum. 

In the National Center for Public Policy and Higher Education’s (2008) Measuring Up 
report, each state was given an A through F grade to indicate performance in an easily 
digestible manner on a variety of indicators tied to institutional quality and effectiveness. 
Included in this report was an indicator on student learning; each state earning a grade of 
“Incomplete” due to a lack of commonly accepted student learning outcomes assessments 
across the nation. In the report, Peter Ewell noted that since the release of Measuring 
Up’s first edition in 2000, fewer institutions seem to be committing to national efforts to 
provide comparable, transparent information on learning (National Center for Public 
Policy and Higher Education, 2008). This lack of cooperation, however, is not due to an 
insufficient array of available instruments to assess student learning. Stanford 
University’s National Center for Postsecondary Improvement houses an electronic 
inventory of dozens of instruments commonly used to measure college-level learning 
(National Center for Postsecondary Improvement, 2013). What, then, is stalling progress 
on assessment as an accountability indicator and a quality assurance mechanism? 

The ability for college and university leaders to respond to stakeholder calls for evidence 
of student learning is met with significant contextual considerations. To understand how 
college and university leaders are attempting to respond, if at all, to expectations for 
student learning outcomes involves a deeper look into the steps currently underway to 
collect this evidence. Given the longstanding concern with accountability for student 
learning, however, it is evident that significant barriers impede the ability for or 
willingness of many institutional leaders to gather, report, and use information on student 
learning. 

The Research Problem and Purpose 

The problem is that although the call to assess student learning persists as an 
accountability challenge, there is an insufficient understanding of what steps, if any, 
colleges and universities are taking to gather such evidence. Further, little is known about 
the challenges faced by college and university leaders to address stakeholder expectations 


Spring and Summer 2014 


86 



Journal of Research in Education 


Volume 24, Number 1 


for evidence of student learning. This knowledge is important because it adds narrative 
to institutional efforts to respond to critique and concern expressed by key stakeholder 
groups expressed through several decades. Consequently, the purpose of this study was to 
explore the present status of efforts to assess student-learning outcomes within the 
bachelor’s degree granting institutions in one system of public higher education. Further, 
the purpose of this study was to understand what challenges and criticisms, if any, are 
reported by academic leaders about the effort to produce evidence of student learning. 

The study was guided by the following research questions: 

-What efforts, if any, are institutions presently taking to assess 
and report student-learning outcomes and why? 

-What types of learning outcomes, if any, are institutions trying to 
measure? 

-What challenges and criticisms, if any, currently impede 
institutions’ abilities to gather learning outcome data? 

Method 

The study utilized a multi-site case study design. Semi-structured interviews, field notes, 
and site documents provided information about learning outcome focused initiatives on 
the three baccalaureate degree granting campuses of one system of public higher 
education. 

Research Sites and Population 

University A is a large public research- intensive institution offering bachelor’s, master’s, 
doctoral, and professional degrees. Approximately 21,000 undergraduate students attend 
the institution. Over 300 undergraduate degree programs are offered. University B is a 
mid-size public institution and offers bachelor’s, master’s, and doctoral degrees. 
Approximately 11,400 students attend the institution and nearly 140 undergraduate 
majors are offered. University C is a small to mid-size public, undergraduate and 
graduate-degree granting campus. University C enrolls around 7,500 undergraduate 
students, and offers bachelor’s, Master’s, and doctoral degrees. Universities A and B 
have been in operation for over a century while University C was founded in the early 
20 th century. 

The population from which the researcher sought participation included not only the 
institutional administrators who held official responsibility with learning outcomes 
assessment, but also those who were responsible for accountability to stakeholders. As 
such, the presidents or chancellors and chief academic officers or provosts were eligible 
participants for the study. The researcher then used chaining from these participants to 
connect with participants who were principally responsible for outcomes assessment on 
each campus. The additional personnel were the directors of institutional research, 
college deans, and vice provosts for undergraduate programs. Twelve participants (4 at 
each campus) were interviewed. 
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Sources of Data 

Three sources of data were used to conduct the study: semi-structured interviews, site 
documents, and field notes. Site documents included institutional accreditation reports, 
accountability documents, strategic plans, and others provided to the researcher by the 
participants. Field notes were taken to provide documentation of observations that 
further illustrated the participants’ responses to the interview protocol that could not 
otherwise be recorded. The interview protocol was comprised of three major inquiries: 
assessment instruments, initiatives, and challenges. 

Procedure 

Upon IRB approval, the researcher contacted eligible university administrators for 
participation in the study. Though chief executives served as the lead off participants, the 
researcher was linked to other administrators who held formal responsibility with outcomes 
assessment. In addition, the researcher was connected to academic administrators who 
work closely with faculty and college deans with regard to outcomes assessment. In 
addition to interview data, relevant site documents were obtained through the university 
website and were also gathered upon recommendation of interview participants. Field 
notes were also taken to provide additional thoughts or observations while in the 
university settings with the participants. Data collection continued until no new evidence 
was gathered. 

Data Analysis 

During the initial within-site analysis, the researcher reviewed data and wrote a 
preliminary list of in vivo codes based on the words and text noted in the data sources for 
each campus (Bogdan & Biklen, 2007). After reviewing the data several times and 
modifying the list of initial codes according to similarity, the researcher read through the 
data to assign codes. Upon completion of coding, the principal investigator generated 
patterns unique to each institution to compare and contrast findings between the 
campuses during the between-site analysis. To answer the research questions, the 
researcher reported themes common between the campuses in relation to the three 
research questions (assessment initiatives, types, and challenges). 

Data Trustworthiness 

Member checking and data triangulation were used to ensure trustworthiness of the 
study’s analysis and findings (Guba, 1981; Shenton, 2004; Merriam, 2009; Creswell, 
2009). The researcher inquired with participants to verify the accuracy of the 
interpretations of data collected in the study (Guba, 1981, Merriam, 2009). The 
researcher also gathered multiple data sources through the study, which were used 
collectively to affirm the trustworthiness of findings noted in each as an independent 
source (Shenton, 2004; Creswell, 2009). 
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Results 

An important contextual factor became clear after analyzing the interviews, site 
documents, and field notes: Student learning outcomes assessment on each campus was 
driven primarily by compliance with state law and regional accreditation standards. The 
duty to assess, however, was delegated to particular personnel on the campuses. 

Accreditation standards articulated that each educational program must identify, assess, 
and demonstrate improvement on a clearly articulated set of learner outcomes to show 
commitment to continuous improvement. State law mandated that each public college 
and university assess and report scores on assessments of general education and major- 
specific learning outcomes. At each campus, personnel who managed assessment were 
getting ready to provide student learning data to the state and were in the middle of 
preparing evidence of compliance with standards for their approaching regional 
accreditation reaffirmation visits. Embedded in their efforts to comply with these 
external stakeholders were perspectives on the instruments, initiatives, and challenges 
associated with student learning outcomes assessment. 

Findings Related to Assessment Types 

The case study included an inquiry into what learning, specifically, is being assessed at 
the institutions. Also, the inquiry focused on what instruments are used to produce the 
assessment results. Two findings related to what is being assessed and by what 
instrument were exhibited: general education and major-field testing. The state agency’s 
requirements for outcomes assessment required that, with agency approval of the 
instrument, institutions develop or adopt measures for major and general-education 
learning that provide a numerical comparison to a standardized or historical score. Each 
institution adopted a different general education outcomes assessment instrument and a 
broad array of instruments for major-specific outcomes. Further, some programs were 
exempt from the state’s assessment standards due to a curricular focus that does not yield 
easily to quantitative assessment (i.e. studio art). 

General Education Testing 

To comply with state accountability standards for general education assessment, 
University A used a quantitative instrument to assess critical thinking skills. The 
instrument is a nationally and internationally standardized instrument and is widely used 
as a measure of college-level critical thinking skills. Critical thinking ability is 
delineated in sub-scores with the intended purpose that the results can be used to identify 
possible curricular improvement areas. Students were tested during the senior year. 
University B used a value-added assessment to comply with general education assessment 
standards. Students were tested at the start and completion of the curriculum. The 
instrument is a widely established, standardized instrument offered in multiple-choice 
format. A written essay accompanies the assessment and is used to examine writing and 
analytical reasoning skills. Questions assess students’ aptitudes for reading, writing, 
mathematics reasoning, science-reasoning, and critical thinking. University C offers a 
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different value-added examination than University B, but students, too, are tested prior to 
and at the conclusion of the curriculum. This assessment also examines core 
competencies associated with the general education curriculum as noted in the other 
instruments. Once every five years, each institution is required to report its student 
learning outcomes data to the state agency to comply with accountability standards. To 
incentivize improvement in the results over years, universities are eligible for state 
performance-based funds if the score from the most recent reporting period improved from the 
prior period. 

Major-Field Testing 

Each university developed and adopted a variety of quantitative assessment instruments 
to assess learning outcomes associated with academic programs. At University A, for 
example, the investigator collected and reviewed the Department of History’s state 
agency-approved major field assessment. The history format is a multi-question, true- 
false examination about specific events and individuals noted in the curriculum. 

Standardized major field tests, such as those administered for programmatic 
accreditation, are given to students within departments that seek such accreditation. One 
example cited in the study was accounting, which at University C is accredited by the 
Association to Advance Collegiate Schools of Business (AACSB). As part of the 
accreditation review and reaffirmation process with AACSB, institutions must report 
Certified Public Accountant (CPA) licensure examination pass rates. 

For some academic programs, the departmental faculty members were engaged in 
embedded assessment of learning outcomes to provide additional information not yielded 
from the state-mandated quantitative instruments. For instance, Judy, Associate Provost for 
Undergraduate Feaming at University A, shared: 

In my discipline, we teach a capstone course which is 
intended to bring together all the learning that students 
have done throughout their studies.. .they [students] have 
to present on a media campaign that they have developed. 

They have to demonstrate that they know the material 
through integrating their knowledge into the project. As a 
faculty, we look for student demonstrations of their 
learning in the presentations. One of the things that we do 
as a faculty after the presentations is to sit down together 
and go over our evaluations of the projects and discuss 
where the students did well, and where they did not do so 
well. 

Though assessment was present at each institution, participants expressed that more work 
was needed to get each department involved, to show skeptics the utility of assessing 
outcomes, and to use the data to strengthen teaching and learning. As a result, university- 
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wide initiatives were underway at each institution to improve upon the assessment 
process. 


Assessment Initiatives 

Each campus had built into their institutional strategic plans the priorities to assess 
learning in response to compliance expectations from the state and from the assessment 
standards articulated by the regional accrediting body. Therefore, themes noted were 
working toward compliance and trying to engage in continuous improvement. 

Working toward Compliance 

Though institutions were in full compliance with state assessment regulations, meeting 
these regulations did not translate to fulfillment of the more rigorous standards associated 
with regional accreditation. State standards only required that scores improved from one 
reporting period to another, whereas accreditation compliance meant a much more 
comprehensive approach of documenting, assessing, and planning improvement in 
student learning. For instance, Gerald, Dean of the College of Arts and Sciences at 
University B, shared that “some of the departments in my college do not get as much out 
of the [assessment] experience as others; they aren’t that serious about assessment.” 

In preparation for each university’s accreditation reaffirmation visits, the participants 
noted that, although they will not fully comply with standards articulated by the regional 
accrediting body, little worry was devoted to what this would mean to the university in 
terms of their reaffirmation prospects. For instance, Sharon, University C’s Associate 
Provost who was responsible for undergraduate assessment, reported, “Right now, we’re 
coming up on [our regional accreditation reaffirmation] and we’re going to get dinged on 
our assessment. We do a lot of data gathering; we just don’t do a lot with it...” 

Getting ‘dinged’ meant that although the accrediting agency may raise concern with the 
campus’ efforts to assess student learning, there was little concern that any serious 
consequences would come from the finding. Only getting ‘dinged,’ however, did not 
mean that the participants were willing to concede progress toward compliance with 
accreditation. 

Though the universities may be unable to fully address the accreditation standard in time 
for their upcoming reaffirmation visits, participants articulated that moving step by step 
toward the standard was an ongoing priority. On each site, the participants stated that the 
immediate first steps are to ensure that each program has a clearly articulated set of 
outcomes. Then, the campus’ assessment leaders plan to have each program develop 
corresponding instruments and a plan for how the data will be collected. To achieve 
these goals, the participants were busy integrating assessment into already established 
quality assurance processes. As Provost at University A, for instance, Allison stated her 
priorities in relation to getting all of the departments engaged in assessment: 
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I’ve been fine-tuning our academic program review process. 

We’re building student learning outcomes into the program 
reviews we conduct internally, which our institution requires 
every five years. Once we do that, our student learning 
outcomes assessment process will be built into the 
accreditation process. 

At University C, the process involved implementing a new centralized database to 
provide greater collaboration between departments and university administrators in its 
established program review process. The database uses a standard structure for 
departments to enter the learning outcomes, assessment plans, instruments, results, and 
plans for improvement of academic programs. From there, Lynn, Vice Provost for 
Undergraduate Programs at the institution states that we “now have a common structure 
in which to plug the data. Now it’s easy to track departments and say, “Well, you don’t 
have anything in there you need.”” The ability to make campus-wide steps forward and 
to provide continuous guidance and support were viewed as critical steps toward fulfilling 
accreditation standards. 

Trying to Engage in Continuous Improvement 

Continuous improvement in teaching and learning was also a work in progress. For 
instance, University A was in the midst of a campus-wide effort to tie learning 
outcomes assessment data into improvement of the general education curriculum. The 
participants shared that this process involved gathering faculty members who teach 
general education courses together to define the competencies associated with the 
curriculum. That work group is led by Judy, the Vice Provost for Undergraduate 
Learning, who shared that, “we are right in the middle of the process of making some 
changes to our general education program for undergraduate students. One of the charges 
to the committee was to be very intentional about what our learner outcomes are for 
general education, how we can assess those, and how we can use the assessments to go 
through an improvement process.” In the report released by this work group were some 
general competencies that students should gain as a result of the general education 
curriculum. Up to this point, the effort had not tied specific courses to the competencies 
or provided an overview of how assessment might be used to improve curricula. 

On each campus, the ability to design assessment systems was a key area of focus in 
trying to engage in continuous improvement. But in attempting to move forward, 
participants shared that two key challenges encountered were to determine which 
instruments to use and, in turn, to develop a plan based on the type of information yielded 
by the results. The quantitative instruments used to comply with state assessment 
standards were met with criticism by faculty in terms of the ability of instruments to yield 
data that could target specific areas of improvement. For instance, John, University B’s 
Director of Institutional Research, shared that “We get faculty who look at the 
questionnaires we administer [for compliance] and say, “Well, this doesn’t represent 
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what I teach.”” In response, John asks, “Well, what would?” He’s also taken leadership 
on an institution-wide effort to provide answers to this question within each academic 
program, but not all units have agreed to participate. 

At University B, the effort to get programs involved in the continuous improvement goal 
have thus far resulted in push for embedded assessment, which Susan, the university’s 
Provost described as “a process where faculty take the work already submitted by students 
in coursework and re-evaluate it for competencies in the curriculum.” According to the 
participants, this type of information provides clarity about what, if anything, should be 
addressed that a number may be unable to discern with credibility and confidence. The 
participants also felt that embedded assessment provided a means for faculty to link 
identified outcomes to specific courses, thereby targeting improvement areas within the 
curricula. 

One illustration of how embedded assessment was cited as a useful resource for 
strengthening efforts to continuously improve program delivery was in the Department of 
Music. Lynn, Vice Provost for Undergraduate Programs at University C, reported that: 

Embedded assessment allows us.. .to look at whether the 
interpretation of the piece is appropriate for the time 
period... It’s subjective, but the faculty have been able to 
develop a way to examine student performance because 
they are qualified in their areas so they can judge. 

The example provided shows how universities are adapting to the expectation that 
assessment lead to continuous improvement, but it also illustrates a divide between the 
call for comparable quantitative data to evidence learning and the expectation that such 
data be used to guide improvement. But, from the participants’ perspectives, it was a 
priority for them to make progress by doing what worked to address accountability 
expectations and strategic goals. However, there was no clear timeline identified for 
when the universities would achieve their strategic plan goals for institution-wide 
assessment. To meet this goal, key challenges would need to be addressed. 

Challenges to Reaching Assessment Goals 

In moving their campuses forward, the study participants encountered challenges that 
impeded the ability to achieve institutional goals to assess student learning and to use the 
data to guide improvement. Namely, the themes related to these challenges were 
resistance based on established practices and concern with assessment decision utility. In 
turn, before efforts could be made to strengthen assessment systems in response to 
institutional goals and compliance standards, these participants shared that much time 
and effort is currently spent addressing criticism, building trust, and gaining support for 
the value of assessment. 
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Resistance Based on Established Practices 

Participants stated that their efforts to assess and report student learning outcomes were 
met with resistance from some academic units, particularly where the faculty viewed 
student learning outcomes as unnecessary or redundant to grades as an indicator of 
performance, and, as a result, of little importance when it comes to using the data to 
guide performance. Many of the participants, like Richard, Dean of University C’s 
College of Business, for example, are often asked what value assessment brings that 
grades do not when attempting to work with programs that do not currently assess 
outcomes. 

In response to the value that assessment data bring to informing the conversation of 
improving teaching and learning, Allison, Provost at University A, shared, “If I have 
assessment data, I can say look, a Classics graduate from this institution should be able to 
do these 10 things, and, doggone it, they can,” instead of, “Well, this person got a 3.5 
GPA in X number of classes.”” Allison’s point is that assessment provides an organized 
crosswalk between what students should be learning and, by linking these outcomes to 
the courses in which the outcomes are taught, evidence can be generated to target 
improvements and to demonstrate performance to stakeholders. Grades and grade point 
averages, in contrast, fail to indicate where students may not be learning expected 
outcomes of students. 

In addition to grades and grade point averages, the participants reported that some 
colleagues resisted assessment because, not having familiarity with the practice, they 
worried about its use and intent. Richard, Dean of University C’s College of Business, 
shared that faculty members are often distrustful of how the assessment data will be used. 
He reported, “It is difficult sometimes convincing faculty members that the data will not 
be used against any particular faculty member, and, of course, we’re not gathering the 
data in order to get rid of faculty member X, and the convincing process takes a while.” 

In response to this concern in his college, Richard assembled a group of faculty across 
the departments to serve in a collaborative manner and to act as liaisons back to the 
faculty to help demonstrate how data are used. In addition, the faculty help communicate 
to Richard how assessment is being used to improve teaching and learning. As a result, 
assessment goals were being met within the college and faculty had a clearer 
understanding of the intent of the assessment process. 

Concern with Assessment Decision Utility 

Participants encountered and perceived concern with the decision utility of assessment, 
also described as the ability for the assessments to yield data that could truly measure 
learning or guide improvement. Bill, University C’s Chancellor, expressed concern that 
the college experience was larger than “a litany of assessment instruments” could gauge. 
He reflected: 
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I’m asked to tell stakeholders all the time about what 
transpired within our students during college and I don’t 
know. They [our students] started years ago and a lot has 
happened to them and not just during college. Those who 
make laws or policy think we produce a product or service 
- that it can be measured and if we can’t show we’re doing 
it then we have to get rid of what we’re doing or change it. 

Another challenge was that of student motivation to take assessment seriously, 
particularly where examinations did not directly affect students based on their 
performance. John, University B’s Director of Institutional Research, shared that he often 
receives completed tests where students fill out the bubbles in various shapes or in a 
Christmas tree pattern, thereby raising questions about how seriously the instruments can 
be taken in making judgments about student learning. 

Participants shared concern with the ability to draw conclusions about where 
improvements can be targeted due to broad overlap in the competencies reinforced in the 
curriculum. At University A, Judy reported, for instance, “We report the data [on general 
education testing] and show comparisons across colleges, but that’s not assessment. 
That’s testing.” The distinction is that data should indicate where, specifically, the 
learning is or is not taking place and should provide useful evidence. The general 
education examination provides statistical means that indicate performance relative to 
other colleges within the institution or to other institutions across the country. However, 
the participants shared that comparable data falls short of being able to guide curricular 
improvement. 

Though participants encountered and recognized the limitations of assessment data, 
particularly in relation to accountability expectations, they also work to overcome 
resistance as a means to achieve institutional quality assurance goals for continuous 
improvement. Those responsible for leading student learning outcomes assessment at 
each campus had engaged colleagues through initiatives to gather evidence that could 
yield improvement-oriented results. As a result, many departments recently started using 
embedded assessment where faculty use assignments from coursework to evaluate 
whether students leam key competencies. 

At the universities, however, efforts to address concerns were a work in progress that still 
faced criticism and challenges to institution-wide commitment. Not all departments 
adopted methods that addressed concerns regarding decision utility. Though progress 
had been made, participants still reported that more work is needed to fully implement 
assessment processes that responded to institutional goals for quality assurance in 
teaching and learning. 


Discussion 

What is stalling progress on outcomes assessment? Consistent with Ewell’s (2001) 
essay on the role of accreditation, regional accreditation was cited as a major factor 
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driving assessment on each campus. But the participants shared that not complying 
with assessment standards raised little concern, citing that they are likely to ‘get 
dinged’ for not being in full compliance. Without serious reprimand for compliance 
failure, institutional leaders and curriculum leaders have little motivation to take the 
practice of outcomes assessment seriously. Will it take the threat or actual loss of 
regional accreditation to more heavily incentivize learning outcomes assessment? Up to 
this point, the reinforcement of standards has not yielded satisfactory results at prompting 
many institutions to prioritize learning outcomes assessment. 

Despite the wide variety of instruments used to assess major-specific learning on each 
campus, and further recognizing the ability for each institution in this study to choose 
from a litany of general education and major-specific assessment instruments, the issue 
of decision utility still persisted as a challenge to the institution-wide establishment of 
assessment practices. Participants from each campus reported concern with the ability of 
assessment instruments, particularly quantitative measures, to yield results that are useful 
at guiding curricular improvement or at reflecting performance to stakeholders 
accurately. Though participants were working to steer the campus forward with 
assessment, their reports only reflected partial, incomplete engagement across the 
academic units. Could ‘engagement lag’ be a major reason why national efforts over 
several decades have thus far failed to produce widespread commitment on the part of 
colleges and universities to respond to calls for evidence (National Governors 
Association, 1986; Spellings Commission, 2006; Ewell, 2008)? 

As was mentioned earlier, the presence of student learning outcomes assessment is not a 
new phenomenon. Yet, in response to accountability standards, why do instruments not 
viewed as capable of producing useful results still get used? With no shortage of 
assessment instruments or approaches, it is unclear why college and university 
assessment leaders would continue to use instruments and approaches that do not yield 
meaningful, credible results among campus faculty (National Center for Postsecondary 
Improvement, 2013). Participants were committed to compliance with standards 
articulated by stakeholders in government and accreditation. But outcomes assessment is 
not just about compliance or accountability. If a divide exists between what is viewed as 
credible by stakeholders and what is viewed as useful by faculty, then campus-wide 
initiatives should extend beyond compliance and focus also on improvement. With 
regard to general education assessment, this was not observed on each campus, and 
differing engagement with academic units was reported on devising approaches 
that responded to criticism about major-specific assessment. 

Although the dialogue up to this point has principally focused on learning outcomes 
assessment in relation to external accountability pressure, this attention is not to suggest 
that institutions have completely failed to respond to expectations for evidence of student 
learning and continuous improvement (National Institute for Learning Outcomes 
Assessment, 2013). To this end, it is also important to consider how stakeholders 
themselves fit into the picture. Why, despite evidence of progress on student learning 
outcomes assessment, does there seem to be a lack of recognition by external 
stakeholders on the efforts of colleges and universities to respond to expectations for 
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accountability in learning? To what extent do expectations held by state leaders align 
with those in external accreditation? Perhaps colleges and universities are caught in an 
accountability paradox where unflattering evidence of performance weighs significantly 
more heavily than achievements. Added to this paradox is the magnitude of higher 
education’s stakeholder audience whose expectations on performance and mission 
priorities may often exist without consensus (Bogue & Hall, 2012). 

Participants also shared that although they encountered resistance from some faculty and 
departmental leaders with regard to student learning outcomes assessment, they were also 
able to make small steps toward achieving institution-wide commitment by engaging 
colleagues in dialogue about what would give them evidence needed to improve teaching 
and learning. This study suggests that if assessment remains as an accountability 
priority, and if leaders can demonstrate value to colleagues, perhaps slow and steady 
progress can be made toward achieving institution-wide commitment to accountability 
for student learning through the practice of assessment. However, with widespread and 
persistent public interest in learning outcome evidence, combined with a continued 
resistance noted at each campus, it is unclear when or if institution-wide learning 
accountability will be attained. 
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